{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Bayesian Program Learning\n",
    "\n",
    "When humans make inferences about unseen data, they make use of strong prior knowledge (or inductive bias) about related events they've seen, heard, touched or experienced. For example, a child after playing with a dog a few times sees cat for the first time would immediately know it's related to dog and not with the food they eat like pasta.  Humans, as opposed to machine learning models, don't need thousands of examples of cats to 'learn' that category once they have already learned to recognize the dog. \n",
    "\n",
    "\n",
    "One way humans acquire this prior knowledge is by their innate capability of ‘learning to learn’. This notion of learning to learn is closely related to ‘transfer-learning’ and ‘multi-task learning’ in machine learning terminology. The terms refer to ways in which learning of new tasks or concepts can be accelerated by previous or parallel learning of new tasks.\n",
    "\n",
    "\n",
    "\n",
    "Generative models are probabilistic models that aim to bridge the gap between human learning and machine learning. These models aim to learn high-level abstract features from parts of an object and apply those learned features to new but similar object categories. \n",
    "\n",
    "\n",
    "In this chapter, we will study how these generative models are built, what it means to have prior knowledge, how to frame the prior knowledge in mathematical terms, how to learn high-level concepts from few objects (learning), and how to combine this newly learned knowledge with prior knowledge in making meaningful decisions about new objects (inference). \n",
    "\n",
    "The following topics will be covered in this chapter:\n",
    "\n",
    "1. Bayesian Learning\n",
    "2. Directed Graphical Models\n",
    "3. Overview of Probabilistic methods \n",
    "4. Bayesian Program Learning\n",
    "5. Discriminative K-Shot Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"Example.png\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview of Probabilistic Models\n",
    "Humans' conceptual learning tends to differ from machine learning by two major aspects. Consider an example of handwritten digits from a large vocabulary in the below figure. \n",
    "\n",
    "First, people tend to learn meaningful information about objects, for example, object boundaries from just one or a few examples and classify them with high accuracy (Fig(i)). On the other hand, machine learning algorithms need tons of data, especially deep learning models that have achieved human-level performance on tasks like object recognition and speech recognition. \n",
    "\n",
    "\n",
    "Second, humans learn a vast majority of functions from just one example, for example, creating new characters (Fig(ii)), decomposing objects/characters into various parts and relations (Fig(iii)) and develop new meaningful concepts/characters (Fig(iv)) from existing knowledge about concepts. On the contrary deep learning models either require special loss functions and architectures for each task which is usually not practical owing to very limited labeled data available for the task.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "How do people tend to learn such richer, robust representations of the objects and concepts from just one example?\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "The learning theory states that more data (not less) is required to learn more complicated models that generalize well. But, humans tend to learn far richer representations that generalize extremely well from highly sparse data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discriminative K Shot Learning\n",
    "\n",
    "There have been several recent k-shot learning techniques that mimic the k-shot learning task by episodic training on simulated k-shot tasks. In 2003, Bakker and Heskes introduced a general probabilistic framework for k-shot learning where all the tasks share a common feature extractor but have a respective linear classifier with just a few task-specific parameters.\n",
    "\n",
    "\n",
    "The probabilistic method to k-shot learning discussed here is very similar to the one introduced by Bakker and Heskes. This method solves the classification task (for images) by learning a probabilistic model from very little data.  The idea is to use a powerful neural network which learns robust features from large supervised data and combine it with the probabilistic model. The weights of the final layer of the neural net act as data which are used to regularize the weights at k-shot time in a Bayesian fashion.\n",
    "\n",
    "\n",
    "The learning framework comprises four phases, namely:\n",
    "\n",
    "1. Representation learning\n",
    "2. Concept learning\n",
    "3. k-shot learning\n",
    "4. k-shot testing\n",
    "\n",
    "The framework with its four phases is shown in Fig below. They are discussed more formally in the following sub-sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"4phases.png\" width=\"500\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### For more details, refer chapter 5 of 'Hands on One-Shot Learning' Book "
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
